Real-time Hallucination Simulation and Sonification through User-led Development of an iPad Augmented Reality Performance

The simulation of visual hallucinations has multiple applications. We present a new approach to hallucination simulation, developed initially for a performance but which proved to have uses to sufferers of certain types of hallucinations. The system allows real-time visual expression, using iPad augmented reality. The system was developed focusing on the visual symptoms of Palinopsia, experienced by the first author. It also allows the hallucinations to be converted into sound through visual sonification. Although no formal experimentation was, we report on a number of unsolicited informal responses to the simulator from palinopsia sufferers and a charity. Introduction Palinopsia is a visual symptom involving trails in the visual field [1], as shown in Figure 1. In addition to this it can lead to very strong after-images [2], similar to when anyone stares at the sun and looks away – but this can be much stronger for Palinopsics and can be caused by objects which are not particularly bright – for example a door. One interesting aspect of Palinopsia is that it can occur in individuals who show no other ill effects. Palinopsia is a symptom rather than a condition. Oliver Sacks the author and neuroscientist, has had the symptoms of Palinopsia [3]. The academic study of Palinopsia is still in the early stages. Scientists are divided in their terminology, using alternative terms such as Polyopia and Visual Perservation as well [4]. They also have different ideas about the causes of the condition. One possibility is a reduction of inhibition function in certain neurons in the visual cortex [5]. This could both explain how the after-image of a moving object is not cancelled out more firmly, and why the evolving patterns seem to come from nowhere, perhaps from under-supressed random firing of cortical neurons. There have also been interesting initial results concerning a common “visual trail rate” of 15-20Hz [6]. Fig. 1. Visual Trails in Palinopsia for a downward moving hand For some the condition of Palinopsia is debilitating, for others they fail to realise the images are not real, and for many it is a lonely condition they are afraid to reveal to anyone. Oliver Sacks estimates that up to 90% of people with the hallucination condition Charles Bonnet syndrome do not mention their hallucinations to others [7]. Charles Bonnet syndrome is experienced by a number of people who go blind, creating very specific, in some cases cartoon-like, hallucinations. The lack of understanding of Palinopsia could perhaps be aided by tools to enable Palinopsics to express their visual hallucinations more clearly to others. Software that can enable some of these people to express and discuss their visual disturbances with a close friend, or perhaps a therapist or medical professional they are working with, may be beneficial. It could help medical professionals and neuroscience researchers to gain a deeper insight into visualising precisely what the subjects are seeing. Simulators and Augmented Reality Static hallucination simulators are used to help people express exactly what hallucinations they have. There has been some work in the use Virtual Reality to model hallucinations to help in teaching people about schizrophrenia [8] and the pharmaceutical company Janssen Pharmaceuticals have actually developed a system for laptop and stereo glasses to simulate hallucinations [9] for training. But neither of these systems are truly interactive, or based on real-time augmented reality. Augmented reality is a method of combining live camera data on a smartphone or tablet with real-time generated images, both appearing on the screen simultaneously [10]. There are many applications available involving this, including some designed to help rehabilitate people with physical [11] or cognitive [12] disabilities. There have also been augmented reality systems to help visually impaired people navigate more easily [13]. What is more rare are applications which manipulate the visual field to simulate what people are subjectively seeing. This is the application which was developed here, which has been labeled the “Halluciphone”.